Virtual work of expression within a virtual environment

ABSTRACT

A system and method for displaying a virtual three-dimensional environment, including: displaying at least a portion of the virtual three-dimensional environment in a head-mounted display, where the head-mounted display includes a retinal tracking device; enabling a user to interact with the virtual three-dimensional environment; displaying a virtual representation of a written work within the virtual three-dimensional environment in the head-mounted display; enabling the user to interact with the virtual representation of the written work; tracking the user&#39;s reading position in the representation of the written work via the retinal tracking device; determining the content of the representation of the written book at the user&#39;s reading position; and modifying the virtual three-dimensional environment based at least in part on the content at the user&#39;s reading position.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/252,706, filed Jan. 20, 2019, and titled, “A VIRTUAL WORK OFEXPRESSION WITHIN A VIRTUAL ENVIRONMENT,” currently pending, whichclaims priority to U.S. patent application Ser. No. 15/587,128, filedMay 4, 2017, and titled, “A VIRTUAL WORK OF EXPRESSION WITHIN A VIRTUALENVIRONMENT”, which claims priority to U.S. application Ser. No.15/007,162, filed on Jan. 26, 2016, and titled “VIRTUAL WORK OFEXPRESSION WITHIN A VIRTUAL ENVIRONMENT,” which is abandoned, and claimspriority to U.S. Provisional Application No. 62/107,313, filed Jan. 23,2015 and titled, “INTERCONNECTED ANALYSIS OF A VIRTUAL WORK OFEXPRESSION,” which is expired, and also U.S. Provisional Application No.62/109,533, filed Jan. 29, 2015, and titled, “A VIRTUAL WORK OFEXPRESSION WITHIN A VIRTUAL ENVIRONMENT,” which is expired, and thecontents of which are incorporated by reference in their entireties.

BACKGROUND

The history of man is very much a history of imagination. The humanimagination is what has allowed our success as a species. Each new placeexplored, every bold step taken and every invention discovered requiredfirst an equally bold step of creative vision.

The earliest evidence of human creativity pre-date that of recordedhistory itself. From simple paintings on a cave wall to the oraltradition of passing down tales of wonder and heroic deeds from onegeneration to the next. From that very moment when a human being wrotehis name on a piece of parchment to that day Winter day in 1439 when thefirst piece of paper passed through a printing press.

The way we tell stories and the technologies we use to convey them havechanged drastically over the millennia but the core emotional reasonsbehind telling those stories remains the same, which is to haveexperiences beyond the confines offered by of our mortal limitations. Wecreate our characters and the worlds they inhabit as a way to livevicariously through these characters.

That desire to have experiences beyond the ones available in our own areliterally hard coded as deeply as our DNA. What else is the act ofdreaming but an unconscious way to simulate experiences beyond our ownlives. Yet despite all the advances in storytelling and technologyprovided by the advent of motion picture, video games, and theater, itcontinues to be the book that is second only to our dreams in spurringour imaginations. Books allow us to deeply delve into other worlds andthey are the most insightful way to experience and share our stories andthe story of humanity.

When this thirst for knowledge and creative expression integral to beinghuman is connected to revolutionary and disruptive changes happening inthe field of computing, a picture begins to immerse. Advances areoccurring in disparate technological fields which may at first blushseem completely separate having no obvious connection.

Some of technologies such as deep learning and machine learning haveuntil now been solely the domain of well-funded research groups,multinational companies, and supercomputers.

Machine learning and deep learning technologies such as thosespearheaded by Google Inc. and IBM through their Watson® computerrecently played successive games of Jeopardy and won against some ofthose most talented human players in the world.

This was once thought unthinkable although it was once equallyunthinkable that a computer would win a game of checkers against a humanopponent. Less than 10 years later Gary Kasparov, the world's championlost a game of chess against IBM's Deep Blue®.

Machine learning technologies have a goal of receiving disparate sourcesof information and aggregating those sources, analyzing them to gainbroad insights that can be used in a multitude of ways. They have theability to learn and improve on their ability without the direct aid ofa human operator and so can make better conclusions about the input theyare given.

Other blooming and quickly advancing technologies such as “VirtualReality” were once expected to spearhead the computing revolution onlyto be seen as a technological flop of the late 20th century. Since thattime however many of the core technologies required for realisticimmersive experiences have continued to advance unabated. Computingpower has seen dramatic increases year after year. Display technologieshave advanced in both their definition and the ability for screens to beminiaturized.

Input devices such as head tracking technologies have gained a degree ofaccuracy and a lack of latency and the price of all these technologiesare finally reaching a price point which is accessible for averageconsumers.

Other supposedly unrelated base technologies are advancing at a rapidpace. Wearable tracking devices, originally created to track a user'sdaily fitness habits are already developing to become health trackingdevices which monitor an array of bodily signs from a user's heartrhythms to skin resistance providing medical doctors a complete insightinto a person's current health state.

EEG headbands that track brainwave states, movement trackingtechnologies such as accelerometers, gyroscopes, barometers and GPSreceivers, multi touch sensors and any number of other sensingtechnologies have grown tiny enough to fit in a cell phone, all in thecourse of just a few years.

Light tracking and camera motion tracking systems are just starting toscratch the surface of their applicability to video gaming. Alreadyavailable to consumers in forms such as the PlayStation Camera andKinect camera use both the visible and invisible spectrum of light totrack out movements and physical actions and even our facialexpressions.

Some of these technologies are already seeing broad adoption in thedaily lives of consumers, others have yet to see mass adoption but willin the short term start to make head ways into our everyday lives.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present disclosure will be more fullyunderstood with reference to the following detailed description whentaken in conjunction with the accompanying figures, wherein:

FIG. 1 shows a head-mounted display showing a rendered virtualenvironment, according to embodiments of the invention;

FIG. 2 shows a book selected on an external device, according toembodiments of the invention;

FIG. 3 shows a book as a page is turned, and the book is moved inmultiple dimensions within a virtual space, according to embodiments ofthe invention; and

FIG. 4 shows a head mounted display that enables a user to receive avirtual environment including a book inside of the virtual world,according to embodiments of the invention.

DETAILED DESCRIPTION

Herein is a method and system describing a platform which is broadlyapplicable to a great number of industries among them, but not limitedto books, journalism, education & art.

The platform described provides methods of rendering and displaying avirtual three-dimensional environment. Within that environment, avirtual object is displayed wherein the object or work has a purpose of,or at least enables, the sharing, recording or receiving of creativeexpressions or communication.

These inputs are then analyzed in combination or relation to gathereddata and information about the virtual work.

The embodiments of the platform are as follows:

Rendering a three-dimensional virtual environment that a system user orhis avatar may move around in and interact with.

FIG. 1 shows a head mounted display FIG. 1. 101 displaying a renderedvirtual environment FIG. 1. 102. A book as rendered inside of that 3Denvironment F1. 103. Both the book and the environment being on that isdisplayed in three dimensions w F1. 104.

The term “virtual environment” should be understood to consist of“virtual reality”

“augmented reality”

We will refer to the virtual object variously as a “virtual object,“work”, “virtual work”, “book”, “virtual book” among other definitions,work of expression, work of literature, stories or story content.

FIG. 3 shows a book as a page is turned 301, and the book is moved inmultiple dimensions within a virtual space 302, 303, 304.

While many different terms may be used to refer to the virtual objectwithin this document but they are referring to the same thing.

That is, any object which may enable creative expression orcommunication.

It may be a book but anything fulfilling the role of expressingcreativity or communication may easily fulfill the role of a book. Thatis, any object which can enable the sharing, recording, transmitting orreceiving of content such as creative expressions or communications.

Examples of virtual works consist of newspapers, periodicals, comicbooks, painted canvas or even a single piece of paper.

The content within the book variously referred to as transmissions,communications, content, expressions or expressive content or storydata.

Creative expressions and communications enabled by the object consist ofbut are not limited to written, oral, visual, audible, drawn,illustrated, typed, animated, or any other method of creative expressionand or communication known to human beings.

The virtual work may take many forms. These range from a reproduction ofan analog object, such as a paper book to modern technologicalreplacements and equivalents, such as an electronic book.

It may also be a virtual reproduction of an object that would notnormally fulfill the role of a work of expression such as a wall. Onethat is only rarely acknowledged to play this role, such as beach sand,a foggy window or a dirty car.

The work could be a completely novel object of various shapes anddimensions rather than a reproduction as long as it fulfills the statedrole. Furthermore, the expression or communication may exist within theenvironment, as the environment or outside of the confines of a specificobject within the environment.

In some embodiments the object has the same features, qualities, andcapabilities as its real-world counterpart. In other embodiments, theobjects are able to act in ways not consistent with, or havecapabilities or qualities out of the ordinary for real world objects.

An example embodiment might be a reproduced paper book with displayproperties, UI properties, multimedia and sound reproductioncapabilities not seen in real world paper books.

In either case, any object within the virtual world that is made tofulfill a work of expression will just as easily fulfill the role.

The work might allow a system user to create his own expressive contentwithin the work or this functionality may be disabled, depending on thework. In addition, the work may variously be filled with expressivecontent from an author or another creator upon receiving it within thesystem, it may be partially filled with content or completely blank.

An important note is that the system is not at all limited a virtualbook, and functions just as well in embodiments where a real book isused in place of a virtual object.

In the example of a virtual book, the book is reproduced to allownumerous interactions, while in an embodiment using a real world object,any interactions available to such an object, analog or electronic wouldbe tracked by numerous methods.

The following embodiment will provide more detail of one such workingembodiment.

The system provides a method for tracking, recognize, and analyze systemuser actions though various input and tracking methods through amultitude of input devices.

In this embodiment the tracking of these interactions will consist ofmonitoring the movements of the book within a 3D space, tracking asystem user's reading position otherwise referred to as positional data,vis-à-vis the virtual object's content.

Some of these devices are consumer ready such as the Microsoft Kinect®and others are on the short- and long-term horizons of the changingtechnology landscape.

In such a preferred embodiment a system user would interface with avirtual environment through the use of a head mounted display, an IRtracking camera, wearable haptic clothing, and a capacitive touchwalking platform. When used in combination, these systems would allow asystem user to move around with the environment and objects within it inways similar to the real world. A system user would then be able torecognize or select a book within the environment and begin interactingwith it. It should be known that while this embodiment is preferred, anyembodiments described throughout this disclosure are also equally goodoptions.

The look, feel, and available interactions of this virtual reproductionare very close to that of a physical paper book made of matter that youmight find in a Barnes and Noble or in a library.

All of the interactions available to a human being, all the ways theatomic matter of a real book could be interacted with in the real worldare potentially available to the book within this environment.

With the right output hardware, a system user could even smell the freshink and crisp paper of the book. Unlike with the objects in the realworld however, the interactions and abilities here can go beyond theeveryday possibilities, these works may be augmented with capabilitiesbeyond what is usual or possible for the object's real-world physicalcounterpart.

A book is a stand in for simplicity of description but a book could justas easily be replaced with any object or device that has a purpose ofenabling creative expression and/or communication.

A system user will be able to enter this environment either directly asmight be possible using augmented reality displays or through the use ofan avatar as might be usual for a head mounted display.

FIG. 1 shows a virtual book being moved and its pages being turned justas a book would in the real world. A system user could turn the pages ofthe virtual book 101. Pick up the book and move it around within thedimensions of the virtual environment 102, 103, 104. They may scan thepages with their eye 105, read the book's words, view its illustrationsand use the book in a similar way as they would a book in the physicalworld.

In this embodiment the virtual book's movement and location within thevirtual environment may be tracked with the addition of reading relatedpositional data and a system user's interactions through the book'scontent.

In either case the goal in this embodiment is to monitor the system useras he reads and interacts with the virtual book. The system will track asystem user's interactions to determine the reading position within thevirtual paper book.

The system will use any data available to it to gain detailed insightsinto the location a system user is currently perceiving. Locations aretracked relative to broad swaths of content such as chapters orspecifically such as a single word or group of words.

Additionally, a predictive analysis based on reading speed could be usedto determine the reading position and content that a system user will bereading at a given chronological time in the future.

Input data such as this is gathered and processed by a CPU for analysis.

Explanation of Inputs and Outputs.

Inputs

Inputs are gathered from one or more system user(s), avatar(s), virtualenvironment(s), virtual object(s) and expression(s). Any information,interactions or combination thereof is said to be the input.

The system tracks such information, actions and interactions aboutvirtual elements and or real world elements objects and runs theseinputs through a process composed of monitoring inputs, analyzing thoseinputs and producing output based on those analysis.

These inputs include any qualities, quantities, interactions or anyother information about the virtual work.

The resulting output of this combined analysis is used to create ortransform the nature of and content of the virtual environment, one ormore system users' avatars, one or more environments, or the virtualobject itself.

Input data also consist of any information gathered about the currentstate, actions and interactions of system user(s) & avatar(s) bothindividually and socially in relation to the work and its content.

These data inputs may be gathered in the real world through inputdevices tracking a person using the system, that system user's currentstate and actions, and in the virtual environment by detailed trackingof elements of the environment including its avatars and the virtualenvironment.

Outputs

Output data consist of any changes caused to any aspect of the virtualenvironment based on the analyzed input.

It encompasses any displayed output in the real world, in the case ofmixed reality,

It encompassed physical changes into the real world through feedbackdevices available.

Examples of output data might consist of changes to the avatars,characters, settings, locations, displayed educational or medical dataor any other changes caused to the virtual environment.

Normandy

The following example embodiment and any below may occur with acombination of fully virtual environments or mixed reality environments.These are not mutually exclusive and while one system may havecapabilities that the other does not, they can each provide an equalexperience in their own way, and can be run in both embodiments.

The following is a hypothetical but entirely possible preferred workingembodiment as might be put into practice.

The following description is a hypothetical and uses a hypothetical“system user”. Like a drawing, this and other “example” embodiments aresimply a more approachable way to explain the features of the system inan easy to understand and approachable manner, but should not be takento limit the system in any way.

It does however clearly explain the possibilities of the system inaction. It is just one specific such example of how the systemconsisting of a near infinite multitude of such examples that might bechosen.

In this example, a system user has a head mounted display on his head.He might be sitting in a 3D rendered room, at a 3D rendered desk.

He selects a tome close to his heart, by a well-known author about thesecond world war. The moment he selects the work from the UI, theenvironment around his desk changes to the beaches of Normandy, France,Jun. 6, 1944.

At the same time, the book is rendered on the virtual desk in front ofhim.

As the system user picks up the book, the clothes of his avatar changeat that very instant to that of a US soldier, upon inspection of the dogtag around his neck, he sees that his character's name is Private FirstClass Joey Larson.

As the system user flips the book open and turns to the first word ofthe first chapter, he looks to the distant waters, he sees the U boatsapproaching as explained in the content he is currently reading. Theallied invasion force quickly approaching the beach.

His action of selecting the book and interacting with it has caused thesystem to analyze the contents of the book, track the system user'sreading position within it by analyzing the viewing direction of thesystem user's eyes. The system then cultivated a number of onlinesources including Wikipedia.com, Google.com, sites about World War IIinformation about World War II documentaries and then do an aggregatedanalysis of the information determined the contents of the sceneinformation and generated scene data that the system determined likelyaligned with the analyzed data.

It then chronologically progressed the scene in synchronization with thesystem user's current reading position.

As the system user continues to read the U-boats reach the shore. Thesystem user is given a choice in the form of a system user interfaceprojected from the book floating within the space of the environment. Toexperience and interact with the battle or to continue reading and tosee the battle from the perspective of the spectator.

Apartment

The following description should not be taken to limit the system in anyway. It is just one specific such example of how the system consistingof a multitude of such examples that might be chosen.

In this example, a system user has a head mounted display on his head.He might be sitting in a 3D rendered room, at a 3D rendered desk.

He selects a tome close to his heart, by a well-known author about thesecond world war. The moment he selects the work from the UI, theenvironment around his desk changes to the beaches of Normandy, France,Jun. 6, 1944.

At the same time, the book is rendered on the virtual desk in front ofhim.

As the system user picks up the book, the clothes of his avatar changeat that very instant to that of a US soldier, upon inspection of the dogtag around his neck, he sees that his character's name is Private FirstClass Joey Davis.

As the system user flips the book open and turns to the first word ofthe first chapter, he looks to the distant waters, he sees the U boatsapproaching as explained in the content he is currently reading. Theallied invasion force quickly approaching the beach.

His action of selecting the book and interacting with it has caused thesystem to analyze the contents of the book, track the system user'sreading position within it by analyzing the viewing direction of thesystem user's eyes. The system then cultivated a number of onlinesources including Wikipedia.com, Google.com, sites about World War IIinformation about World War II documentaries and then do an aggregatedanalysis of the information determined the contents of the sceneinformation. It then chronologically progressed the scene insynchronization with the system user's current reading position.

As the system user continues to read the U-boats reach the shore. Thesystem user is given the opportunity through a floating system userinterface projected outside of the book. to experience and interact withthe battle or to continue reading and to see the battle from theperspective of the spectator.

As the system user looks up from the book he is reading, he begins tohear, the sounds of fire and pain, the casualties on the battlefield. Hefeels the emotion of that say in a way not possible since hearing thestories his grandfather told him of what it was like to live though theexperience.

He hesitates but looks up only to actually see and hear the battlenearby in all its unfortunate glory. Then as the system user reads on,sadness tearing up in his eyes, he turns to next chapter, suddenly hebegins hearing the sound of percolating coffee,

The book suddenly cut scenes to a different setting. He looks up to seea city apartment, as he continues to read the book explains that he isnow in a New York apartment belonging to Corey Davis, the brother ofPfc. Joey Davis. The system user's avatar now wears the clothes of CoreyDavis and is the character of Corey Davis. Corey hankered to join thewar effort but due to a back injury was decline service by the Army.

The system user notices a character walk in the door of the apartment.The book explains the character to be the father of the two brothers ashe walks in, he says “Hey Cor, you want a coffee?” He hears a voice comefrom his own direction, as if it was his own “sure Dad, how are youdoing”.

As the system user reads his character and the other characters presentin the scene, in this case Corey and his father, speaks and acts thescene in the book word for word. This can be seen in FIG. 4 401 shows ahead mounted display allowing the system user to receive the virtualenvironment. FIG. 4 402 shows a book inside of the virtual world beingread by a system user. FIG. 4 403 shows the words being read by thesystem user chronologically. FIG. 4 404 shows the character speaking andacting in chronological sync with the system user reading the book. As asystem user reads, a character on scene is acting out and speaking thescenes in real time.

Davis's words are spoken in line with the reading position.

Events described in the book, such as Mr. Davis going to get a pot ofcoffee in bringing a cup to Corey, is also presented in real time inchronological order as they are being read.

In this way a system user's reading of a book is, played out like amovie word for word. The system user almost literally is living in theworld of the book.

It is possible to have the scenes acted out in any unsynchronized formas well or any combination. Some embodiments would have a system userwould read an entire chapter and then see the content or scene data ofthat entire chapter played out at once.

The system is not solely limited to completed works, it also allows thecreation of works.

In such embodiments the system might use an input device to recognizethe input of creative expressions in the real world. Other systems maytrack inputs a system user makes relative to the virtual world.

For example, let's take a novelist who is writing a book. In thisembodiment and physical input device is used, in this case, taking theform of close approximation of an antique typewriter.

The novelist is writing a book about piracy on the high seas in theearly 1800s.

As the system is able through its input and analysis of the writingstyle and the essence of the information being written, it begins tochange the displayed environment around the system user.

In this case, the system has recognized that the story takes place whenthe 1850s to ship called the Ristoon.

The system through inputs from the system user, and gathering andprocessing thousands of pages of information from outside sources about.Such information might consist of, how such ships were built, of whatmaterials, how they might have appeared from outside and theirinteriors, the sounds they made whilst sailing, how the crews dress ortalked.

The may track the novelist as he types his book and once the systemthrough its analysis of the type information and has come to aconceptual understanding of that content. This conceptual data orprofile data and the resulting scenery data does not come from onespecific piece of gathered information.

Rather the system uses numerous clues based on observed connectionswithin the work.

The system does not gather information about the ship in isolation butthrough an intelligent analysis connects different parts of the writtenword. It might look at the era or year described in the work.

It might notice that the work takes place on earth in the 1800s, thatthe work is one of nonfiction;

The system through gathering information from any number of outsidesources.

In a way similar to IBM's Watson and other dynamic learning computersare able to come up with answers based on a culled database searchresulting in more accuracy than their human counterparts.

This system will function on any intelligent computing system, and mighteven be able run on “Watson” itself but specifically uses a system withone or more processors.

With similar or superior capabilities is also able to come to aconceptual understanding of data and answers to that data in a wayhumans do not have the capacity to achieve.

It does not speak of some alternative reality. That is a biography, thatthe biography is about a specific captain of that era.

It might determine thousands of connections like this in addition toadjectives describe.

The system also has applicability for museums. In such an embodiment aholographic laser display system, or other system providing a mixedreality experience.

Let's take an example of a museum. As spectators walk through the museumthey will come across various exhibits.

This may be anything from dinosaurs to cars from computing or any otherinformation that that might be found in a museum such as the Smithsonianfor example.

As these visitors read the placards, or monitors displaying informationabout each exhibit, the system though some physical input device maydisplay a three-dimensional representation either in front of orcompletely encompassing the visitor.

This information could be visual information related to the exhibit.Whether it be educational or just to provide additional entertainment.

An example might be visitors of the natural history Museum at theSmithsonian®.

In a related hypothetical working embodiment, one or more uses gather atan exhibit about the Jurassic period. The system displays a visualenvironment that completely fills these visitors' field of view in 360°.This might display the prehistoric environment.

In this embodiment a display showing an animated progression related tothe exhibit might be synchronized with the three-dimensionalenvironment.

This may show not just one environment, but the environment could changefrom one era to another in progression. This also shows how the systemis not restricted to a single environment, environments could changecompletely or could move to different environments based on the analysisand synchronization process.

A Use Case for the Medical Industry.

In a medical emergency room, might walk up to a patient chart outside ofthe emergency room. As the doctor looks at the chart, a 3Drepresentation of a recent CT scan appears in midair, this allows thedoctor to make a superior determination about the reason for thepatient's hospitalization

Another example may be a surgeon operating on a patient. Before theoperation begins as the operation is occurring, the doctor may pick upor view a paper binder with operating instructions that has beenprepared ahead of time. These types of instructions are generally use bysurgeons to map out the exact needs of the patient during the operation.As the attending physician views the surgical instructions, a 3Drepresentation of the human anatomy might be shown in relation to thesurgical instructions. The floating 3D representation shows the exactstructures within the human body or specific visual information to helpguide the doctor though his surgery.

The system has broad applicability in educational settings.

For example, a situation where a classroom is learning about the solarsystem.

In this embodiment a mixed reality display may be used.

This could be anything from traditional augmented reality glasses, to aholographic or laser display. In any case the students would interactwith a book within their class or other educational setting.

A teacher might guide the students through their textbook whichdescribes the orbits of the various planets.

As the teacher points to a blackboard, which in this circumstance actsas the object or work, the system displays three-dimensional graphicrepresentation of the sun, and visual representation of the planets andthe orbiting tracks that the planets follow, may appear in midair.

As the teacher begins to speak in depth about the planet Mars, thegraphical representation a three-dimensional of the planet. As theteacher might speak thin atmosphere of the planet, she could then pointto information on the blackboard about habitability of the surfaceitself.

As he touches a pointing wand at the blackboard location which speaksabout the surface, a light detecting device detects the movement of thewarrant to the work.

This changes the three-dimensional representation in size and scope. Therounded object, which was just seconds before only about two feet indiameter, begins to zoom into the surface, the object growing toencompass the entire room.

The 3D depiction of the planet is no longer an isolated object floatingwithin the room but now encompasses the entire room. From the viewingposition of those within the classroom, they appear to be on the surfaceof Mars. Rather than a floating object, the depiction shows them asinhabiting the surface itself, and they could see the ground themountains and the desolate red landscapes of the red planet.

This is just one such example, and should not be seen to limit thesystem to this example, however it does show when a working system mightlook like within an educational setting.

Another Working embodiment, may be a history professor talking about therise and fall of ancient Rome. The class goes through their text bookswhich go over a short synopsis of each era of roman history.

A 3D representation of Emperor Caesar guides the class page by page textpage by page though the text book. This computer controlled avatar goespoint to point, page by page, from the Rome's early founding by themythical Remus and Romulus, to the adoption of Christianity and thecreation of the Byzantine Empire. Throughout this process scenery data,such as characters, settings and other visual information changes. Thisin effect takes the students through a guided audiovisual journey ofancient Rome, providing and enhance learning experience of learninghistory by “being there”.

And environment related to the field of journalism might function inthis manner.

A newsreader receives his daily copy of the New York Times. This may bea physical version of the New York Times that the system tracks as inputor a virtual object.

In either case, the man as in all examples herein, may use mixedreality, virtual reality or a combination thereof.

As the news reader progresses through the international stories of theday, the system could transport the reader through its output scenerydata, showing the locations and events of the daily story.

Unlike television news where a system user would only see that new storyas might be explained by the television anchor.

This would allow a reader to experience the events and locations ofthose news stories in three dimensions as if the system user was there.

This has great application for enhancing and experiencing the worldaround us, while still allowing us to experience the news through thetraditional medium of a newspaper or magazine.

The Artist

Let's take an artist whose creative expression is painting. This couldbe any style of art, and the campus could be real or virtual.

In either case the system would be able to analyze the painted work, ofthe landscaping painting.

As the painting goes from a blank canvas to something resembling a “workof art” the system might track the artistic input or expression createdby person, in this case visual.

As the painting is in the process of coming together the systemsynchronizes the painting on the canvas and through its intelligentanalysis of the visual properties of the painted work, displays athree-dimensional version of the painting around the artist.

In this way the artist is placed with in a realized of three-dimensionalrepresentation of his own art.

Tracked Input

What is input data? It is a myriad of information and data points arecollected about one or more people using the system, any information orelements within the virtual environment.

It is outside information from various sources about various topicswhether specifically related to a system user, the virtual object, it'scontent or expressions or anything other information feed into thesystem for analysis.

Some number of data point about a system user's virtual avatar orrepresentation within the virtual environment are also collected andanalyzed by a CPU.

The data points are gathered by various input, tracking and detectionsystems. Data points could be any information about one or more peoplescurrent or past actions, interactions or state of being.

These include but are not limited to tracking of current, past orpredicted information related to a person's state within the real orvirtual worlds.

The following are more detailed examples about the type of data that maybe considered “input data”. It is important to understand that these aresimply a cross section of possible embodiments and should not be takento limit the scope of this disclosure to just these examples.

Interactions with the Work

Any information about a system user's avatar that may be tracked can beincluding;

Achievements or the current level or status of the avatar.

Actions or interactions of an avatar or system user within the virtualenvironment, such as generated scenery information, objects, or computercontrolled avatars.

Actions, interactions, or communications between one or more otheravatars within the environment or between multiple separate or connectedenvironments within the system.

Any tracked information but without being limited to the above may begathered, processed and analyzed. Crucially however the aboveinformation is not gathered in isolation.

It is gathered in relation to any possible combination of interactionsbetween work(s), system user(s), avatar(s) or environment(s).

Any information about a system user's avatar that may be tracked can beincluding;

Achievements or the current level or status of the avatar.

Actions or interactions of an avatar or system user within the virtualenvironment, such as generated scenery information, objects, or computercontrolled avatars.

Actions, interactions, or communications between one or more otheravatars within the environment or between multiple separate or connectedenvironments within the system.

The combination might consist of the tracked system user informationabove and that may be tracked in relation to the work itself or thescene information generated from it. This may consist of;

Interactions with the Work—Virtual World Inputs

Alternatively, it could be tracked signals of a system user or hisavatar within the virtual environment, such as;

A system user or his avatar's interactions with the work within thevirtual world or interactions with other system users. This may include;

Moving the work either within the virtual environment or between virtualenvironments. Adding or removing, receiving or transmitting any contentto or from the work.

Giving or receiving the work to or from another person or avatar.

Operating or controlling the work in some manner. Using or not using orretrieving information to or from the work for any given amount of time.

Combining the work with other works or splitting the works intodifferent parts.

A system user or his avatar's location or movements within the virtualenvironment.

A system user or his avatar's location or movements within the virtualenvironment.

A system user or avatars interactions with one another.

A system user or his avatar's location or movements within the virtualenvironment.

A system user or his avatar's location or movements within the virtualenvironment.

A system user or his avatar's location or movements within the virtualenvironment.

A system user or his avatar's interactions with scene informationderived from the virtual world.

This section talks about what is going on around and to the book. Ittakes the book's perspective to speak about what I felt was ambiguousabout my first description when I said the words “AND IN RELATION TO THEBOOK.”

This was a bit of a revelation in explaining it correctly.

Input data is not limited to system user data, or information about thatuser's interactions a work either directly or through and avatar.

Has the work been moved to a specific location within the environment?

How many people are interacting with the work?

Are these avatars close to the work or far away?

Is the work currently in a default setting, an unrelated setting, or ina setting of its own content?

Other embodiments may track input data relative to the expressivecontent and provide scenery data out of synchronization, or synchronizedbut not as a user is progressing with any specific piece of content.

An example might be content shown about a chapter at the end of achapter, or at the end of a book itself. Or even information related tothe expressive content of more than one works.

It might be information about a series a multitude of journals, notes,movies, comic books, newspaper articles, printed pages or any number ofexpressive works.

The system is not limited to analyzing the input data of one piece ofexpressive content and it may use more than one expressive work eitherin combination or separately, in parallel, or in sequence, either syncedor chronologically separated.

This input data is processed through a number of algorithms, analysistechniques and computing methods.

In many embodiments the aim of this process said to be discovering theessence of the expressive content or information. Although this may beachieved through different computing methods for depending on theenvironment, it uses among other techniques, semiotic analysis.

In some embodiments the system might go through a process of gatheringraw input information and process it with an aim of forming connectionsbetween disparate and varied sources of information. From this thesystem gathers concepts and comes to an understanding of the data andhow the various inputs and their content related with each other.

A preferred embodiment processes raw input data from internal andexternal sources. This raw input data is processed to find visible orhidden connections between these disparate sets of data.

These connections are then reduced to hypothetical concepts whichprovide a subtle and detailed view of the information. These conclusionsshall be known herein as conceptual profiles.

Several computing methods are available to the system.

While any and all traditional computing methods are available to thesystem, the system may use and combination of traditional andnontraditional communing methods.

Nontraditional methods such as but not limited to deep learning, machinelearning, and dynamic learning will be used in the goal of understandingthe concepts and essence of expressive content. While traditionalmethods may assist in this, many nontraditional methods, algorithms andanalysis techniques are available to obtain this goal of gathering anunderstanding of expressive content.

Gathered information may go beyond simply input data as described above.Many disparate sources of information may be accessed and passed to theanalysis process along with the input data. This includes informationcontained on internal databases, external databases, whether private orpublicly available. Online sources

consist of online encyclopedias, research databases, patents, governmentdatabases, dictionaries, current and historical periodicals and/ornewspapers. Just about any website or online source that a human beingwould be able to gather information from is included, with the additionto any source that might require a computer to access and/or understand.Local databases may include any information stored locally or on offlinesources, including information stored on magnetic storage, non-volatilestorage or other means of storage medium that is accepted as statutorysubject matter.

These analyses include but are not limited to the following. Althoughthese are stated specifically, different embodiments may use manydifferent analysis and processing techniques not listed here.

Natural Language Processing

An analysis most of us have become familiar with through interactingwith,—speaking—to our smartphones. Natural language processing is ananalysis technique aimed at understanding human language. This couldhelp the system understand written language within the content. Whilethe systems abilities would not in any way be limited to processingwritten content, NLP analysis could be used to process any writtencontent to understand words in a way similar to the way a human wouldread and comprehend it.

Pattern Discovery

A method of finding patterns within information, could be used to locatepatterns within the written information or story to help come to aconclusion about its contents.

Predictive Modeling

Predictive modeling may be required in embodiments where a system useris himself Is providing the opportunity to create new content. Someembodiments speak about a blank or partially completed work where asystem user can at his own creative expression within the work. In theseembodiments, predictive modeling can be used, possibly in concert withthe other analysis techniques to not only gain an understanding of thecontent of an expressive work but to predict information about theconstitution and concepts of content that is likely to be created.

Hypothesis Generation & Evaluation

This gathered information then uses among other techniques, knowledgerepresentation, and hypothesis generation.

These techniques are uniquely able to analyze many disparate sourcesconsisting of data such as from online sources, internal databases,input data. The system then uses this disparate information to generatea hypothesis about the nature of the expressive content.

These and other computing methods and analysis techniques are used todetermine big picture and little picture guesses about the essentialnature of the expressive content, it's story and any specific elementsor constructs therein.

It's important to note that not all embodiments go through this analysisprocess, and some skip it entirely relying on a process of pre-generated1:1 synchronization between the work and the outputted scenery data.

In this description we term the output data as any changes,modifications, transformations or the creation of any new informationwithin the environment based on these hypotheses.

In this way, the conceptual profiles may be deemed part of the analysisprocess, or part of the output process.

Even though they are a type of output as that word generally used, evenso, it's not what “stage” they reside in that is important, it's thatthese conceptual profiles allow output to occur.

These profiles are more or less a way for the system to categorize itsfindings about what the story, the expressive work or the communicationis about.

Concept profiles may be determinations or hypothesis related to anynumber of story elements, or in the case of nonfictional work, anynumber of informational attributes or concepts.

These include but crucially are in no way limited to the followingexamples;

Visual Character Profile: This is any information about a character'sphysical appearance, manner of dress, or any information related to it.

Audible Character Profile: This is any information about the soundsspeech method of communication or any other audible related informationabout a character.

Communication Profile: This is any information that is communicated byor about a character within the story

Character Profile: The actions the character takes, whether it beinformation about a character's movements location actions, speed ofmovement, thoughts, words spoken, or any other active or passive actionsof a character.

Character interactions Profile: are any communications or interactionsbetween characters.

Character Event Profile: Is any events that occur to the characterswithin the story.

Character Time Profile: Is any information about character actions orinteractions within a specific chronological period of time.

Setting information such as:

Local Profile: locale information may be broad consisting of a universe,galaxy, planet, country, state, region, city, or town. It could also bemore specific locations such as a neighborhood, street, building or asingle, room or any other locale of any dimension.

Time of year Profile: Time of year relates to any time within 365 days ayear, or dependent on the planet in question, a year may constitutesomething different. In the case of earth, it could be about a specificseason holiday and anniversary of some important date, such as PearlHarbor or any other yearly related data.

Time of day Profile: this could be anything about how the story unfoldsminute to minute, hourly or within a given 24-hour period.

Elapsed time Profile: Time related data may be about time from one eventto any other given event.

Mood Profile: This may be anything such as temperature, lighting, moodor any other environmental emotional data about the content.

Geography Profile: this is any natural data about the setting and wherethe story takes place. Any scenery data including information aboutwhether the scene is flat or mountainous, loss with greenery and treesor an empty desert, a harsh climate, a warm paradise.

Man-made or synthetic objects: This will be any information aboutanything within the scene that is not natural and created by biologicallife. This would include obvious man-made objects such as buildings,cars, changes to natural environment such as pavement or any otherindicators of synthetically created objects and locations.

Historical Profiles: Historical time is any information related to aknown or fictional date time or place. “Include any information relatedto man-made objects known for specific period of time. This includescharacter related information, such as style of dress for the period,manner of speech, architecture for the period or any other story relatedinformation that is somehow connected with the historical period intime.

Plot Profiles: This is any information about a story's narrative and theevents within the story. This naturally encompasses all other types ofdata we speak

about above. Any information about characters, setting but also thechronological order that the events occur in, the dramatic structure,Literary elements and narrative structures both linear and nonlinear.

Direction and Predicted Modeling: Is any information gathered andpredictions made about the direction of a story and how it might playout.

Informational Profiles: In the case of nonfictional, educational andother expressive or communicative works which do not rely on storydriven elements, the system will gather an informational profile.Examples of informational profiles consist of the following;

Medical-Related Profile: Any data related to the field of medicine orfitness, whether it be a patient's personal medical data or history,some data related to medical research, elements and objects related tomedicine, such as the structure of the human body, DNA, surgicalinformation or any other information related to the field of medicine.This information may be called from disparate sources and provided to asurgeon or Dr. Within a mixed or virtual environment.

Educational Related Profile; Any information related to content directedwithin the field of education. This includes information that might befound in textbooks, whether they be related to science or scientificobjects or information, literature language, mathematical, historical,data such as historical events, or any other information that may betaught within an educational setting whether it be first grade oruniversity level.

Commerce Related Profiles: Include any information related to commerce,financial, self-help, leadership and any other information related tothe field of business, Stock market, commodities, and the depiction ofother commerce related information. The analysis may take disparateinformation about market trends and other disparate information aboutpast current and future trends, trades and actors within the market aspart of this profile.

Statistical Data; Is any information related to the field of statistics.

Journalistic data; is any information about world events, what is goingon in the world, how disparate events relate to each other, predictionsof future events, is written by a journalist within a newspaper or otherjournalistic medium.

The system is not limited to the above data profiles, and many otherprofiles may relate to any number of different industries or types ofcontent and expression information whether visual, writing or oral,story, informational or communicative.

In addition, the generated profiles are merely an organizationalconstruct, in short, it is the way for the system to categorizedifferent types of input data, come to conclusions about such data andoutput information into recognizable categories.

This might be helpful in the system analysis but also provides a way fora human operator to more easily understand the hypothesis.

Even so, profiles as described do not isolate the ability of the systemto analyze collected data or come to determinations based on that anyinput data it has available to it. They should not be seen as limitingeither the type of data gathered or hypothesis concluded from that data.Some embodiments may work just as well without a system of organizedprofiles.

The system may aggregate data in different ways, and come to conclusionsabout information not just within a given profile but across profiles aswell. Other working embodiments might use entirely differentcategorization methods, analysis methods, or computing methods.

In either situation, the system is able to gather different kinds ofinformation from disparate sources, and using the aggregated data tocome to a recognition or understanding of the essence of the expressivecontent or communication therein.

The profiles gathered are not simply a bunch of randomly aggregatedinformation. While some of the information may be randomly gathered forone reason or another, the system uses one or more computing methods oranalysis methods either to a hypothesis and understanding of the contentof the virtual object.

This gathered information and any hypothesis generated may be performedby one or more systems.

These may be performed on a single CPU or multiple CPU's and the meansfor producing or processing the conceptual data is varied.

This may be in the form of an artificial neural network, clusteranalysis, hierarchical clustering outlier detection, associationlearning or any other number of

techniques, analysis and computing methods as a means for processing theinput data.

The hypothesis and determinations can be both very broad or veryspecific depending on the input data.

This hypothesis consists of any understanding, comprehension,prediction, determination of the content, the connections between thecontent and how input data and gathered internal and external sources ofinformation relate and interaction with one another.

The system then uses these hypotheses to determine which data to create,modify or remove from any and all elements of virtual environment(s),avatar(s) and virtual object(s) or physical object(s), to any feedbackdevices or outputting display devices.

Output is considered any information the system produces from theanalysis process.

and are deemed herein variously as “scenery data”, “scene data”, “sceneinformation”, “feedback data”, “display output” depending on the type ormethod of output.

Changes caused by the analysis of inputs.

System output is any information the system produces through analysis ofuser input data. This information may be pre-rendered, procedural orgenerative data produced through this analysis.

This analysis does not need be processed using machine learning orpowerful computer systems.

Embodiments that don't use such techniques should not be consideredinferior, they simply have different applicability in one embodiment maywork better for one user and another with another user depending inrequirements and/or the situation, industry or use case.

Some embodiments use pre-generated output data such as those created byan author to match his work.

This would allow a creator to have full “creative control” over howsystem users interact with the work and how “output data” is rendered inrelation to which input data.

For example, in an embodiment where the system renders sand display'svisual output data that an author or a publisher created to fit specificsections of the work.

An example embodiment might be visual or audible output datasynchronized to chronological reading position.

This embodiment would be closer to that of an interactive book or a flipbook in that the content is already generated by a human artist/creator.These creations may then be synchronized to specific user actions werelinked via tags, OCR detection or other cues to the work of creativeexpression.

In yet another embodiment, the author or creator may be able to allow acreator to control generative or procedural output.

This might be superior for an author or publisher who wanted to enablesystem users to use their expressive content within the system, but didnot want to put in the extra time or effort to create specific content.

In some embodiments a human would pre-program various conditions, thismight be helpful in situations where the processing speed to dointelligent machine learning analysis are not available to a user.

Some embodiments would allow an author or publisher the ability to setany number analysis modifications, or output conditions, providing somehuman control over the outputted generated work. The output data, howand when the output data is output in relation to the input data.

In either case the way the system gets from input data to output datamay take various paths, from human guided, intelligent analysis, basedon pre-programmed conditions or matching inputs to pre-created content.

The system can handle any of these, a mix of them but is in no waylimited in its method of producing output. These range from traditionalor non-traditional computing techniques analysis and completelyautonomous or with human guidance.

Any information output through the gathering or processing of inputdata, that may be perceived by the 5 senses is deemed “Scenery Data”.

An example of this would be changes produced by the system based on theanalyzed input data, whether that analysis includes hypothesisgeneration or not. Additions, modifications, transformations or deletionof any information within the multi-dimensional virtual environment orto any directly or indirectly connected systems would be consideredscenery data.

Although this goes without saying, it is important to note that inputdata and output data do not need to be the same. Input data may becompletely transformed

or converted to a different form of expression or communication, andpresented in a completely different or novel way.

Just one non limiting example might be the words of a book and theinformation therein might be presented in a completely different manner.Text information might be displayed as audio, audio information asvisual, visual as haptic and even haptic back into written. Anytransformation of one format or context to another and any combinationor mix are considered within this disclosure.

IN addition, information input is not 1:1 with information outputted.The information presented might be an amendment or modification of theoriginal data and or completely new data all related or unreleased dataaltogether.

as displaying one form of content in a completely different format ormethod.

Just one such non limiting example might be one type of data, say forexample written information, displayed in a different format altogetheras in the Mars example previously.

Any sort of substitution or of one type of information or method ofcommunication to another type.

Information “visual, audio or haptic or smell information. Any sort ofcontent transformation to a completely new method. This might besituations where the information is similar to the input information orcontain completely unique or novel content apart from the inputinformation.

This may include but is in no way limited to; Any content orinteractions made available.

The recreation, reproduction, realization, rendering and or display offictional people, objects, settings, world or otherworldly phenomenon,objects or settings.

This need not necessarily be a recreation of things found in the realworld and may be completely novel or imaginative content.

The display information as a moving sequence of events or a narrative,which may occur in chronological or non-chronological order.

The progression or sequence of events could be character movement,speech, interactions, actions, a sequence of information about anyobjects, settings or characters. Any narrative or structure producedthrough computer graphics.

The display or retelling of any form of information whether it isdisplayed statically or dynamically

Any communications passed on or outputted by the system.

Any movie like, or game like production.

The production where a story is told or information conveyed.

Any smell related information that is produced by the system.

Any audible information produced by the system.

Any haptic data or touch data outputted.

Visible and invisible light output

Any audible output

Any spoken words

The display of any movements including the movements of characters,objects or creative expressions

The display of informational content either statically or dynamically.

Anything Rendered as computer graphics

Anything Output as audio or radio waves

Produced as an audio visual combination It may be;

A reproduction of one or more object

Shown as a story that progresses in the environment sequentially ornon-chronologically

Displayed as characters, locations, settings, realized narrativeinformation

Displayed as character movements actions or interactions

Any data related to input data that is produced in an animated state, ora static visual representation

Displayed as artistic information

Displayed as information

An audio soundtrack of musical composition that is related to theexpressive work.

Displayed information about the narrative data

Statistical information

And visual or audible information that is displayed that is related tothe expression content either directly or by extension.

Any User Interface Data

Any interactions made available, such as interactions made available toa system user or interactions made available between system users.

Any interactions enabled or made available to a user and his avatar byany scenery data or through directly and indirectly connected systems.

Any output that a user may experience that is produced by extension ofthe input data

Outputted in these and other forms is known as scenery data.

The effect of this is greater than the sum of its parts, it is truly amarvel that is difficult to explain in words the emotional result thesystem is able to provide.

In the case of expressive works with narrative structure, this systemprovides a user the ability to interact with, experience and virtuallylive inside of “the world of the book”.

And to use lay language for a minute, this is the systems number oneaim, to

This might consist of the static display of information, the animateddisplay of information, the progression of a plot in a realized audiovisual manner, narrative or context in which that information isdisplayed.

This output data might be simple like the display of a static object tocomplex life like characters, settings and information.

The system has the capability of displaying characters and settings in away you might experience them in a movie or in the real world.

These characters might speak, communicate, move, act, and interact in avery realistic way.

This may come in the form of fully realized narrative reproductions ofstories, the progression of events related to the expressive work, thecharacters, settings and any information gathered through the input dataand content of the expressive work.

For example, in the case of a book about the Industrial Revolution, asystem may envelop the user in a scene showing workers working in atextile mill, which may then progress to showing the workers protestingfor fair wages.

It may be supplementary content to a movie that is playing in thevirtual environment.

If a user was reading a book about fixing a car, the relevant partsmight be displayed and tools required.

A user that was reading a story about ancient Egypt, and might be ableto see the pyramids as they are built.

An important point is that the system is not limited to showing a 1:1representation of information or content within a book. Some embodimentsmay use a 1:1 synchronization between a system user's progressionvis-à-vis the expressive content.

Chronological synchronization is an option for the system but the systemmay take other forms as well.

Depending on the processing and graphics power allotted, a system userin some embodiments might even find it hard to recognize the differencebetween real life and the created environment.

In some embodiments the use may explore or interact with the virtualenvironment. This result is allowing a user interact with the “world” ofthat story, and the people (characters) and “locations” created throughthe books content.

Let's use the example of a sci-fi novel. A user might be given a promptto stop reading and interact with the data either himself or withseveral other avatars in an “adventure” style mode.

In the case of a post-apocalyptic book about underground bunkers or“silos” as is the case in the recent serialized novel “wool”.

Readers might be able to put the book down and walk around the silo.Visit its various rooms. Speak with the residents of the silo, ask aboutthe events related to the book and get responses about them.

The may interact with this environment with their own avatars or thecharacters themselves, taking on the physical appearance and voicequalities of that character.

This could take many forms.

The system user may interact with the elements of the environment.

A user might be able to interact with one or more builders, who isdressed like, acts like, and speaks like someone of the time period. Theuser may ask questions, have them respond with direct answers.

Furthermore, these inputs, analysis and outputs are not an isolated stepprocess.

Interactions between these steps may produce recursive effects where asystem user or an avatar('s) interactions with the output data, changesthe input data, and by extension the resulting output.

This means that interactions with or changes to outputted scenery databy system users or their avatars relative to the system output mayfeedback into inputs and by extension changing the resulting output.

Let's take the example above where the user is given an opportunity tostop reading and to instead with the scenery data directly.

In the example of a working embodiment where the user is interactingwith the world of the book, and the system user has is controlling theavatar who has taken the place of the “main character”. The system userthrough interactions may actually change and progress the story line.The user may discover an object relevant to the story. They may learn ordiscover information from one of the story's characters. They may changeand object or interact with the character or change the progression ofevents.

In some embodiments these changes the “world” will feed back intochanges in the expressive work.

This is in some ways similar to “make your own adventure” styleproductions, except in this case it's more apt to say it's “live yourown adventure”

A user's interactions with the “world of the book” might end up changingthe book itself.

This is fully relevant for non-narrative embodiments. In thosesituations, interactions with three-dimensional informational constructsdisplayed within the system might change the information within the workproviding that information.

So, in the example of mars in the augmented reality example above, asystem user might be reading information about the history of theplanet.

That user may get up from a chair and point to a specific section of thethree-dimensional representation. As the user goes back to look at histext book, the data within has been update to a detailed description ofthat information.

In addition to interacting with the book, some embodiments will allowthe system user to put the virtual book down, interact with the book orthe characters around him. The system may allow the system user toeither put down the book, and interact solely with the environment for atime. Examples of this would be a pop up to “join the battle”, or “solvethe puzzle” or “race the course” or “fly the plane”.

In this way input data produces output data that produced modified inputdata based on a user's actions.

Output data could be information that is provided to physical feedbackdevices.

Scenery data could be provided to any number of feedback devices withina user environment. This could be smell generators to approximate theenvironment of the system user.

If a user is wearing input devices such as a “haptic suit”, withfeedback capabilities, the system could provide output informationrelated to the scenery data.

A user could literally touch and feel the visual environment and thescenes described within the work.

May interact with characters through touch.

May smell scenery data through a device that outputs smell informationand experience.

Through output to such feedback devices, the user may experience theenvironment with all five senses.

Output data is not limited to a single environment or changes to morethan one separate environments.

Furthermore, it also consists of any output data sent to or collected byany external computing devices, systems, output systems or devices whichmight be directly or indirectly connected with the platform, such assystems that are separate from but connected in some way to the coreplatform.

This connection might take the form of coding API's or other publiclyavailable APIs, which might allow external software companies orproducers of hardware to interact with the system.

Examples of output mediums might be external displays, tablets or mobiledevices, software programs, smart devices, smart lights, connectedspeaker systems, connected A/V systems, or any other system that is ableto output information retrievable through the five senses or may provideinteraction for a human being.

Just one specific non-limiting example might be Philip Smart lights.Let's is a hypothetical example where the scenery data is beingdisplaying as a locale.

Within this locale,

The system may display output information through audio, visual, andother systems.

The system is not limited to being displayed on a single display thetime. The system may make use of a multitude of directly and indirectlyconnected display and feedback systems.

This includes but is not limited to, fixed-display systems such astelevision or computer monitor, head mounted displays, tablet displays,holographic display systems, laser projection systems, clear glassdisplay systems one example being a car window or projection into thehuman eye or through display implants, or any system that is a means forthe projection of information as perceived by a human.

Augmented Reality (AR or MR) is one such possibility. Augmented realityallows the system to intermix the virtual environment and the user'sreal environment. As the word virtual environment is used throughoutthis disclosure, when the system displays a virtual environment withinthe space of a system user's real-world environment, then the real-worldenvironment is considered part and parcel of the virtual environment.

Whereas in the virtual environment the dimensions in which the contentand output information is project is recreated using computer graphics,in the Artificial reality examples, the dimensions of the user'sreal-world environment becomes the construct and structure in which thevirtual environment exists.

In situations where the system uses augmented reality as a means fordisplay, the user will be able to directly interact with the virtualenvironment and the content therein directly with his own body, or withthe aid of other devices within the user's real world.

This provides the added benefit of allowing multiple system users tointeract with one-another in their real-world environments. Two or morepeople in the same physical location, could interact with each other andthe virtual environment through their physical presence.

A novel aspect of this system is the idea of “user is avatar”.

User is avatar is one way the system enhances the immersion of a userinto the virtual environment.

This could come in the form of the projection, display or visualaugmentation over the user's body, accessories, clothes, and evenhandheld objects in relation to input data.

And another example might be somebody taking the place of a maincharacter present within the input data. This system would be able toaugment a user's physical appearance, voice, style of dress or any otherpersonality or visual attributes over the physical body of the personwithin the virtual environment. This concept might be broadly applicableincluding outside of the platform described in this specific disclosure.It would also allow for the display or augmentation of physical objectsthat are being one body or carried by the physical person interactingwith the environment. This user would then be able to see and hearhimself as the main character in the story and so would other peoplethat were perceiving information as produced by the system.

In combining this with multiple users in a physical space as describedabove.

You might have multiple people in a room playing different characters ofa famous book or story or movie. Each character might choose a characterin the story to “play or “be”.

They see themselves and the other people in the physical space with themas those characters.

Any clothing accessories are objects that the characters might pick upand use are also displayed. Less use an example of two medievalswordsmen. The users would see each other and themselves with armor onand swords in their hands, these virtual objects, in this case theswords might be real physical objects, such as a stick, wherein a swordis augmented and depicted in place of the stick or physical object. Thisprocess may completely generate any new objects within the environmentor augment any existing real-world physical objects, making them appearas some other object.

Some non-limiting examples might be a virtual dress up program whichmight allow someone to see what they look like with virtual cosmetics,or virtual clothing.

Any output data that the system may display in a fully virtualenvironment may also be displayed in the example of augmented reality;

The system may variously fill a user's entire field of view with virtualelements. It may alternatively mix-and-match real world or virtualobjects, for example showing some real objects within the user space,while showing other objects as virtual objects.

As explained in the earlier example some real objects may be augmentedto look like virtual objects. This is not limited to handheld objectsand may be other objects within a user's physical space. For example, acouch that system users are sitting on may become a car seat, the scenebecomes the seat of the car and around the couch a car is displayed,with one of the system users acting as a driver and the other acting asa passenger in a moving vehicle. Around the vehicle they may see the carmoving within an animated scene.

In other examples physical devices acting as a means of input into thesystem. If the earlier example of a journalist with a typewriter isused, a system user might be sitting at a real desk with a realtypewriter in front of him typing on a real or virtual piece of paper.In any case the system will be able to keep the typewriter and thesystem users within his or her field of view. Around the typewritereverything in the user's field of view could be a virtually generatedscene in a full 360°.

In the same example, alternatively, the desk shown could be the realdesk or other objects in the room could be real objects within thatuser's real physical space. Any combination of real or physical displayor augmentation is possible in early compatible with the system andplatform.

In no way should the means of display be used to limit anything statedin this disclosure, additionally the system would be able to adapt withand take advantage of the capabilities of any display or feedback systemthat produces output through the use of a processor.

Data Inputs range from system users' physical state when interactingwith the work. Examples of inputs include any information gathered byphysical sensors. These include those detecting body state such as heartrate, mind state such as EEG, location as gathered by GPS, movement andgenetic data as gathered by visible or invisible light sensors,accelerometers, barometers or gyroscopic sensors.

Any information about a system user's avatar that may be tracked suchas;

A myriad of embodiments may be combined. Input data points are collectedabout one or more people through a number of input and output devices.

The data points are gathered by various input, tracking and detectionsystems. Data points could be any information about one or more peoplescurrent or past actions, interactions or state of being in the physicalworld.

Embodiments including;

A listing of all the different things that might make up input.

An embodiment where the input data is a user's heart rate is monitoredin relation to his or her use of the system.

Wherein a means for capturing heart rate data is used. Non limitingexamples for a means for capturing such information might be a fitnesstracker or a portable ECG.

And embodiment where a system user's kinetic movements while interactingwith the system is the input data.

Wherein a non-limiting means for capturing kinetic data may becapacitive touch platforms including those that might be walked on,visible or invisible light sensors and radio wave sensors.

An embodiment where the data input is a system user's eye position wheninteracting with the work or the content inside of it.

Wherein a nonlimiting means for capturing eye position on data is aretinal tracking device, to device that is integrated with a headmounted display, using EEG technology to track the signals of a user'svisual cortex may be on other example of a means for determininglocation viewing location.

And embodiment where input data is a user's brain signals.

Where in a non-limiting means for capturing brain signals might be anECG device that a user wears on their head.

An embodiment with the system tracks one or more users interacting witheach other.

An embodiment where

Input Data May be gathered from a plethora of outside Devices, sources,databases directly or indirectly, via connections and interconnectionswith the system.

This may use an API to allow third-party developers to integrate withthe system, or may be the system interconnecting with outside devicesand online databases to retrieve out.

A multitude of information about the virtual world the elements of itincluding avatar actions, interactions, and even the output data itselfmight be taken into consideration

An embodiment where input data is interactions between avatars.

An embodiment where input data are the past or future achievements orthe current status of one or more avatars.

Information on the state of the and information about the virtualenvironment and its system users in relation to the book are tracked notlimited to the following;

An embodiment where the number of users interacting with the workerworks is considered as input data.

An embodiment where the number of users interacting with the work isinput data

An embodiment Wherein the input data is related to the punitive timesince one of more users has interacted with the work

An embodiment wherein the input data is the current location with anenvironment the work resides.

An embodiment wherein there is multiple environments simultaneouslyexisting. in such an embodiment it may be useful to use informationrelated to which environment the book exists and use that monitoredinformation as input data.

An embodiment wherein the feedback device is a small generator.

Embodiment wherein the feedback device is a connected light system

An embodiment wherein the feedback device is a smart appliance.

Additional capabilities add can be myriad but examples such astransforming a book into a completely different shape or form,projecting system user interfaces from them or allowing unusualinteractions such as the ability to control the book using thoughts,using the object to control other objects or content within theenvironment, displaying different forms of multimedia within the workeven though the reproduction is of a paper book.

The expressions or communications are not stuck inside the one virtualobject. The book may completely separate from the object itself, forexample with the words displaying in a different location or floating inspace within the environment. transmitting the content of one work toanother work, transmitting the object to another virtual world orenvironment, copying the work to create duplicates or sharing the workwith other avatars by copying it

In embodiment one electronic book is the virtual work.

Shopping Experience

In some embodiments, the book would be select by or received from avirtual bookstore. The bookstore might be represented with in thevirtual world itself. In this embodiment a system user would bedisplayed an interface where he could

browse, read and purchased books. This interface could either be 2D orfull 3D interface such as a replication of the real-world bookstore.

In the latter example, a system user would be able to walk around withinthe real-world bookstore, to browse the shelves, to pick a book fromoff-the-shelf, preview and read the book, go over to the cash registerto see the attendant and/or librarian, ask questions, purchase and/orborrow the book and perform other interactions consistent with but notlimited to the kind of interactions available to a human being within areal-world bookstore. Some interactions not available within the realmight be possible such as pointing or pressing though the use of anavatar in a certain location to bring up a floating 2D interface showingsupplementary information.

Information within this floating had or pop up exists within the 3Dshopping environment and can be interacted with while continuing tobrowse the reproduction of the 3D environment. This would provide anenhance experience allowing system users to leave reviews of a chosenbook, to bring up a display showing information from other online andoff-line sources. So, in this manner a system user would be able tonavigate a replicated emulation of a real-world bookstore or libraryinstantly be able to bring up information within this 3D interface thatwould be consistent with the kinds of information available on an onlinebookstore such as Amazon.com or barnesandnoble.com.

Providing a virtual object within the environment. The inceptiveattributes of the 3D environment may be related to the work or mayalternatively be unrelated to it.

Selecting or receiving the book vis-à-vis the virtual environment mightoccur in various ways.

In a situation where a book is selected before the environmentscreation, it is possible for the system to analyze the contents of thework before creating the environment or anytime thereafter.

There is no chicken or an egg issue, they could be separate or connectedfrom the beginning, and either is sufficient for a working platform.

Other options might be starting with a featureless environment with orwithout a work in it, a random work in it, a related environment or anunrelated environment.

Methods of receiving the work might be variously through,

An e-commerce platform either within the virtual environment orexternally of the virtual environment, such as external devices displayssystems and software and online stores.

Note regarding extra figures.

FIG. 2 shows a book selected on an external device, in this fig, andiPad is shown.

FIG. 2 201, FIG. 2 202 shows a book being selected from a personallibrary.

It will be understood that there are numerous modifications of theillustrated embodiments described above which will be readily apparentto one skilled in the art, including any combinations of featuresdisclosed herein that are individually disclosed or claimed herein.These modifications and/or combinations fall within the art to whichthis invention relates and are intended to be within the scope of theclaims, which follow. It is noted, as is conventional, the use of asingular element in a claim is intended to cover one or more of such anelement.

What is claimed is:
 1. A system for displaying an environment, thesystem comprising: a computer processor; a head-mounted displayincluding a tracking device operable to track a user's reading position;and a content engine executing on the computer processor and configuredto: display at least a portion of a virtual three-dimensionalenvironment in the head-mounted display; enable a user to interact withthe virtual three-dimensional environment; display a virtualrepresentation of a written work within the virtual three-dimensionalenvironment in the head-mounted display; enable the user to interactwith the virtual representation of the written work; receive the user'sreading position in the representation of the written work from thetracking device; identify the content of the representation of thewritten book at the user's reading position; and modify the virtualthree-dimensional environment based at least in part on the content atthe user's reading position, wherein modifying includes producing avisual representation of a character described in the written work, andwherein the visual representation of the character speaks at least aportion of the written work corresponding to the user's readingposition.
 2. The system of claim 1, wherein: the content engine isfurther configured to access a set of resources that provide contextualinformation associated with the content of the representation of thewritten book at the user's reading position; and modifying the virtualthree-dimensional environment is based at least in part on thecontextual information.
 3. The system of claim 2, wherein the set ofresources comprises at least one selected from a group consisting of anInternet website, encyclopedia, research database, patent, governmentdatabase, dictionary, periodical, magazine, and newspaper.
 4. The systemof claim 1, wherein producing the visual representation of the characterincludes representing the character acting in chronological sync withthe content of the representation of the written book at the user'sreading position.
 5. The system of claim 1, wherein the character speaksin chronological sync with the content of the representation of thewritten book at the user's reading position.
 6. The system of claim 1,wherein the virtual three-dimensional environment comprises an avatar,wherein the avatar is based on a character included in the written work.7. The system of claim 1, wherein the tracking device comprises at leastone selected from a group consisting of a camera, infrared camera, andEEG sensor.
 8. The system of claim 1, wherein the written work comprisesat least one selected from a group consisting of a book, journal, workof literature, work of expression, sheet music, text document.
 9. Amethod for displaying an environment, the method comprising: displayingat least a portion of a virtual three-dimensional environment in ahead-mounted display, wherein the head-mounted display includes atracking device operable to track a user's reading position; enabling auser to interact with the virtual three-dimensional environment;displaying a virtual representation of a written work within the virtualthree-dimensional environment in the head-mounted display; enabling theuser to interact with the virtual representation of the written work;receiving the user's reading position in the representation of thewritten work from the tracking device; identifying the content of therepresentation of the written book at the user's reading position; andmodifying, by a computer processor, the virtual three-dimensionalenvironment based at least in part on the content at the user's readingposition, wherein modifying includes producing a visual representationof a character described in the written work, and wherein the visualrepresentation of the character speaks at least a portion of the writtenwork corresponding to the user's reading position.
 10. The system ofclaim 9: further comprising accessing a set of resources that providecontextual information associated with the content of the representationof the written book at the user's reading position; and whereinmodifying the virtual three-dimensional environment is based at least inpart on the contextual information.
 11. The system of claim 10, whereinthe set of resources comprises at least one selected from a groupconsisting of an Internet website, encyclopedia, research database,patent, government database, dictionary, periodical, magazine, andnewspaper.
 12. The system of claim 9, wherein producing the visualrepresentation of the character includes representing the characteracting in chronological sync with the content of the representation ofthe written book at the user's reading position.
 13. The system of claim9, wherein the character speaks in chronological sync with the contentof the representation of the written book at the user's readingposition.
 14. The system of claim 9, wherein the virtualthree-dimensional environment comprises an avatar, wherein the avatar isbased on a character included in the written work.
 15. The system ofclaim 9, wherein the tracking device comprises at least one selectedfrom a group consisting of a camera, infrared camera, and EEG sensor.16. The system of claim 9, wherein the written work comprises at leastone selected from a group consisting of a book, journal, work ofliterature, work of expression, sheet music, text document.
 17. Anon-transitory computer-readable storage medium comprising a pluralityof instructions for displaying an environment, the plurality ofinstructions configured to execute on at least one computer processor toenable the at least one computer processor to: display at least aportion of a virtual three-dimensional environment in a head-mounteddisplay, wherein the head-mounted display includes a tracking deviceoperable to track a user's reading position; enable a user to interactwith the virtual three-dimensional environment; display a virtualrepresentation of a written work within the virtual three-dimensionalenvironment in the head-mounted display; enable the user to interactwith the virtual representation of the written work; receive the user'sreading position in the representation of the written work from thetracking device; identify the content of the representation of thewritten book at the user's reading position; and modify the virtualthree-dimensional environment based at least in part on the content atthe user's reading position, wherein modifying includes producing avisual representation of a character described in the written work, andwherein the visual representation of the character speaks at least aportion of the written work corresponding to the user's readingposition.
 18. The non-transitory computer-readable storage medium ofclaim 17, wherein: the plurality of instructions are configured toexecute on the at least one computer processor to further enable the atleast one computer processor to access a set of resources that providecontextual information associated with the content of the representationof the written book at the user's reading position; and modifying thevirtual three-dimensional environment is based at least in part on thecontextual information.
 19. The non-transitory computer-readable storagemedium of claim 17, wherein producing the visual representation of thecharacter includes representing the character acting in chronologicalsync with the content of the representation of the written book at theuser's reading position.
 20. The non-transitory computer-readablestorage medium of claim 17, wherein the character speaks inchronological sync with the content of the representation of the writtenbook at the user's reading position.